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Problem / Background / Context: 

Randomized controlled trials (RCTs) are increasingly used to evaluate programs and 
interventions in order to inform education policy and practice. High quality reports of these 
RCTs are needed for interested readers to understand the rigor of the study, the interventions 
tested, and the context in which the evaluation took place (Mayo-Wilson et ah, 2013). 

To promote high quality study reports, researchers and journal editors have developed reporting 
guidelines (Moher, Schulz, Simera, & Altman, 2010) that highlight key information needed in 
research manuscripts. Reporting guidelines typically consist of reporting standards — 
recommendations about the content that authors should consistently and transparently report — 
that are based on previous research and developed via expert consensus using rigorous, 
systematic, and transparent methodology (Simera, Altman, Moher, Schulz, & Hoey, 2013). 

For example, the Consolidated Standards of Reporting Trials (CONSORT) Statement — and its 
extensions — are the preeminent guidelines for reporting RCTs; they are based on empirical 
evidence and expert consensus about biases related to validity (Schulz, Altman, Moher, for the 
CONSORT Group, 2010). Since its launch in 1996, CONSORT has had a considerable impact in 
the biomedical sciences; numerous reviews in the biomedical literature have shown an 
association between improvements in reporting quality and these guidelines (Turner et ah, 2012). 

While CONSORT guidelines are well-known in the social and behavioral sciences, there is less 
evidence of widespread uptake and implementation in education compared with biomedical 
disciplines, and deficiencies persist in the reporting of intervention trials in education 
(Torgerson, Torgerson, Birks, & Porthouse, 2005). A common explanation is that current 
standards in prominent reporting guidelines are not adequately tailored to RCTs in education and 
related disciplines. For example, researchers have asked for more information related to process 
evaluations, such as intervention theory of change, assessment of intervention mechanisms 
during the trial, and relevant information about the influence of trial context. 

Purpose / Objective / Research Question / Focus of Research: 

Following recommended techniques for guideline development and dissemination (Moher et al., 
2010), this presentation focuses on a structured research programme to develop a reporting 
guideline for intervention RCTs in education and related disciplines. Namely, this presentation 
will overview the reporting quality of RCTs in education, and discuss reporting guidance for 
reporting education intervention RCTs. 

Improvement Initiative / Intervention / Program / Practice: 

An official Extension of the CONSORT Statement for Social and Psychological Interventions 
(CONSORT-SPI) is under development to provide tailored guidance for reporting RCTs in 
education and related fields. This project involves five phases (Montgomery et al., 2013), of 
which three have been completed at the time of this submission: 

(1) a literature review of previous reporting guidelines and the reporting quality of social and 
psychological intervention RCTs (Grant, Mayo-Wilson, Meledenz-Torres, & 
Montgomery, 2013); 
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(2) a modified Delphi process to generate a list of possible items to include in the 
CONSORT Extension; 

(3) a formal consensus meeting to select the reporting items to add to, or modify for, the 
CONSORT-SPI Extension; 

(4) writing of the guideline documents, including an explanation and elaboration (E&E) 
document that will provide detailed advice for each item and examples of good reporting; 

(5) guideline dissemination, with simultaneous publication and endorsement of the guideline 
in multiple journals, endorsement by funding agencies, presentations at conferences and 
other meetings, and a dedicated website that will facilitate feedback about the guideline. 

Setting: 

Previous reporting guidelines were identified using an adapted version of a peer-reviewed 
electronic search strategy (Moher et ah, 201 1). We also searched three registries of reporting 
guidelines: the EQUATOR Network library of identified health research reporting guidelines 
(www. equator- network.org), a recent review on the development and contents of reporting 
guidelines for health research (Moher et ah, 2011), and a systematic review of studies assessing 
the quality of conducting or reporting trials (Dechartres, Charles, Hopewell, Ravaud, & Altman, 
2011). Education intervention RCT reports were found from a hand search of the 2010 Table of 
Contents of the 10 journals in education with the highest impact factors in the ISI Web of 
Knowledge 2010 Journal Citation Reports (JCR) for Social Sciences. 

The modified Delphi process took place online and involved participants from 32 countries. The 
formal consensus meeting took place in Oxford, UK in March 2014 and involved participants 
from four countries. 

Population / Participants / Subjects: 

The review of previous guidelines yielded 19 documents: 6 documents developed by the 
CONSORT Group, 6 documents developed by medical researchers, and 7 documents developed 
by social and behavioral science researchers. The search for education intervention RCTs yielded 
89 RCTs in education journals. 

Participants in the Delphi process were 384 stakeholders in social and psychological intervention 
trials, including 46 education researchers — of whom 19 are editors of education journals. There 
were 3 1 participants at the consensus meeting, of whom 4 specialize in education research. 

Research Design: 

Previous reporting guidelines and reports of education intervention RCTs were identified via the 
literature search outlined above. A reporting guideline had to consist of a published, peer- 
reviewed article that introduced a fonnal, itemised checklist of reporting standards relevant to 
trials of social and psychological interventions. For the review of RCT reporting quality, reports 
had to discuss a randomized experiment of a complex intervention with psychological, social, or 
health outcomes. 

We used an online, modified Delphi process to select and reduce the number of possible 
checklist items for CONSORT-SPI. Two Delphi rounds were used. Participants included 
researchers, journal editors, funders, policy-makers, practitioners, and consumer advocates. They 
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were identified via relevant intervention literature, journal editorial boards, and organisations and 
societies; signing up on our project website or indicated their interest at conferences; or 
recommendations by our International Advisory Group or other Delphi participants. Participants 
were asked to rank the importance of including each item in the CONSORT-SPI Checklist, and 
to provide comments on their rankings. After Round 1, participant responses were summarized 
and reported back to participants, in order to inform their rankings in Round 2. 

We held a formal consensus development conference to finalise the content of the CONSORT- 
SPI Checklist. A selected group of stakeholders met for three days in March 2014 to reach 
consensus on the checklist content. Evidence from literature reviews and the Delphi process were 
presented on the first day, along with chaired discussion of these presentations. The group then 
anonymously voted on checklist content on the second day through chaired discussion. The final 
day involved a review and amendments of the draft checklist decided on the previous day, as 
well as a discussion of the guideline dissemination and implementation strategy. 

Data Collection and Analysis: 

We examined the content of eligible reporting guidelines by compiling reporting items from all 
identified guidelines into a comprehensive, non-redundant, itemised list of items. To assess the 
quality of reporting guideline development, we compared the techniques used by guideline 
developers to recommended techniques (Moher et al., 2010): preliminary work, development of 
the guideline itself, publication, and dissemination activities. To assess the reporting quality of 
identified trials, two reviewers independently assessed whether trial reports adhered to each item 
in our comprehensive list of relevant reporting item. 

Items in the Delphi surveys were arranged under headings that correspond to the conventional 
ordering of sections in a scientific article (i.e., Title, Introduction, Methods, Results, Discussion, 
Other Information). In round one of the survey, participants rated each item on a 10-point Likert 
scale (ranging from 1, not important; to 10, very important to include in the checklist). The 
average score and a measure of disperson were then calculated for each item based on the 
participants’ responses. Participants also had the opportunity to comment on the checklist items 
or to suggest additional items. Comments were analysed to look for common themes on how to 
modify the wording of existing checklist items or on additional checklist items to consider in 
Round 2. In the second round, participants re -ranked items that did not reach consensus, using 
one of the three following options: “Include,” “Exclude,” or “Unsure”. For each item, the 
percentage of participants marking “Include”, “Exclude”, and “Unsure” were counted. 

Comments were once again analysed to look for common themes related to the wording of 
existing checklist items or on the CONSORT-SPI checklist as a whole. 

At the consensus meeting, participants were be led in structured discussions of each item 
proposed for the checklist from the Delphi process. Voting was confidential using anonymous 
ballots to promote honest answers and allow participants to rethink their position if a re-vote was 
needed. 

Findings / Outcomes: 

We identified 19 reporting guidelines that yielded 147 reporting items relevant to social and 
psychological interventions. Social and behavioural science guidelines included 89 standards not 
found in CONSORT guidelines. However, CONSORT guidelines used more recommended 
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techniques for development and dissemination compared to other guidelines. Our review of 
RCTs (n = 89) revealed that many standards were poorly reported, such as identification as a 
randomised trial in titles (1% of education RCTs reported the infonnation) and abstracts (32%); 
information about blinding/masking (11%), sequence generation (18%), and allocation 
concealment (3%); and details about actual delivery of experimental (45%) and control 
interventions (41%), participant uptake (27%), and service environment (25%). Only 3 of 10 
education journals referenced reporting guidelines in “Instructions to Authors.” The Delphi 
process and consensus meeting yielded a checklist of reporting items pertaining to the essential 
content of reports of social and psychological intervention RCTs. Items in this checklist are 
based on empirical evidence of and expert consensus opinion about their importance to RCTs in 
education and related disciplines. An explanation and elaboration (E&E) document will provide 
detailed advice for each reporting item, as well as examples of good reporting from trials in 
psychology and related disciplines. 

Conclusions: 

Existing reporting guidelines have important limitations in content, development, and 
dissemination in the social and behavioral sciences. Important details are routinely missing from 
RCT publications in education, and most leading journals in this area do not ask authors to 
follow reporting standards. 

This CONSORT Extension is an important step toward improving reports of education 
interventions, improving the accuracy, comprehensiveness, and transparency of RCT reports in 
education. Endorsement of the guideline by journals, research funders, and professional 
organizations will facilitate use of and feedback on the guideline, which in turn will help to raise 
the quality of standards in public health research. 
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